This is very nice, thanks!

We'll still have to add a saving step at the end of an evaluation type. For example, saving at the end of loglikelihood evals, or generative ones. (We should be able to safely assume that once the first item of a given type's batch runs, the rest will run fine too since they are sorted by inverse length)

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry, I am not sure I fully follow. Please let me know what changes are necessary.